X-ray image synthesis from ct images for training nodule detection systems

ABSTRACT

Systems and methods for generating synthesized medical images for training a machine learning based network are provided. An input medical image in a first modality is received. The input medical image comprises a nodule region for each of one or more nodules and a remaining region. The input medical image comprises an annotation for each of the one or more nodules. A synthesized medical image in a second modality is generated from the input medical image. The synthesized medical image comprises the annotation for each of the one or more nodules. A synthesized nodule image of each of the nodule regions and synthesized remaining image of the remaining region are generated in the second modality. It is determined whether each particular nodule of the one or more nodules is visible in the synthesized medical image based on at least one of the synthesized nodule image for the particular nodule and the synthesized remaining image. In response to determining that at least one nodule of the one or more nodules is not visible in the synthesized medical image, the annotation for the at least one not visible nodule is removed from the synthesized nodule image.

TECHNICAL FIELD

The present invention relates generally to x-ray image synthesis from CT(computed tomography) images, and in particular to x-ray image synthesisfrom CT images for training nodule detection systems and other machinelearning based systems.

BACKGROUND

The early detection of lung disease is critical for successfullytreatment. One important task for the early detection of lung disease ispulmonary nodule detection in chest x-ray images. In the currentclinical practice, pulmonary nodule detection is performed manually by aradiologist reading x-ray images. However, the ambiguity of the x-rayimages caused by the 2D projection of x-ray beams results in a highnumber of missed nodules. In addition, with the increasing number ofx-ray acquisitions and the general shortage of radiologists, the amountof time allotted for radiologists to read x-ray images is limited,further contributing to the high number of missed nodules.

Recently, machine learning based nodule detection systems have beenproposed for automatically detecting pulmonary nodules in chest x-rayimages. Such machine learning based nodule detection systems are trainedusing large scale chest x-ray datasets with annotated nodule locations.However, the ambiguity of x-ray images also makes the task of annotatingthe nodules in large scale datasets of x-ray images challenging.Further, to ensure accurate annotations, consensus results obtained fromexperienced radiologists is required, significantly increasing the timeand cost required to obtain such datasets.

BRIEF SUMMARY OF THE INVENTION

In accordance with one or more embodiments, systems and methods forgenerating synthesized medical images for training a machine learningbased network are provided. An input medical image in a first modalityis received. The input medical image comprises a nodule region for eachof one or more nodules and a remaining region. The input medical imagecomprises an annotation for each of the one or more nodules. Asynthesized medical image in a second modality is generated from theinput medical image. The synthesized medical image comprises theannotation for each of the one or more nodules. A synthesized noduleimage of each of the nodule regions and synthesized remaining image ofthe remaining region are generated in the second modality. It isdetermined whether each particular nodule of the one or more nodules isvisible in the synthesized medical image based on at least one of thesynthesized nodule image for the particular nodule and the synthesizedremaining image. In response to determining that at least one nodule ofthe one or more nodules is not visible in the synthesized medical image,the annotation for the at least one not visible nodule is removed fromthe synthesized nodule image.

In one embodiment, it is determined whether each of the one or morenodules is visible in the synthesized medical image by comparing anintensity of pixels in the synthesized remaining image to a threshold.In one embodiment, it is determined whether each of the one or morenodules is visible in the synthesized medical image by determining achange between an intensity at a center of the particular nodule in thesynthesized remaining image and an intensity at the center of theparticular nodule in the synthesized nodule image and comparing theintensity change to a threshold. In one embodiment, it is determinedwhether each of the one or more nodules is visible in the synthesizedmedical image by determining a change between an average intensity ofpixels within half of a radius of the particular nodule in thesynthesized remaining image and an average intensity of pixels withinhalf of the radius of the particular nodule in the synthesized noduleimage and comparing the change to a threshold.

In one embodiment, the synthesized medical image may be generated byincreasing an intensity of pixels in the synthesized nodule images by ahighlighting factor. In one embodiment, the synthesized medical imagemay be generated by positioning a previously acquired synthesized noduleimage in the synthesized medical image. In one embodiment, thesynthesized medical image may be generated by rotating the input medicalimage and generating the synthesized medical image from the rotatedinput medical image.

In one embodiment, a machine learning based network may be trained basedon the synthesized medical image.

In one embedment, the first modality is computed tomography and thesecond modality is x-ray.

These and other advantages of the invention will be apparent to those ofordinary skill in the art by reference to the following detaileddescription and the accompanying drawings.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 shows a method for generating synthesized medical images fortraining a machine learning based system, in accordance with one or moreembodiments;

FIG. 2A shows an exemplary computed tomography medical image, inaccordance with one or more embodiments;

FIG. 2B shows an exemplary synthesized x-ray medical image, inaccordance with one or more embodiments;

FIG. 3 shows exemplary synthesized images, in accordance with one ormore embodiments;

FIG. 4 shows a comparison of nodules highlighted with differenthighlighting factors, in accordance with one or more embodiments;

FIG. 5 shows exemplary images of hidden nodules hidden behind anatomicalstructures, in accordance with one or more embodiments;

FIG. 6 shows exemplary images of hidden nodules hidden due to therelative intensity of the nodules, in accordance with one or moreembodiments; and

FIG. 7 shows a high-level block diagram of a computer.

DETAILED DESCRIPTION

The present invention generally relates to methods and systems for x-rayimage synthesis from CT images for training nodule detection systems.Embodiments of the present invention are described herein to give avisual understanding of such methods and systems. A digital image isoften composed of digital representations of one or more objects (orshapes). The digital representation of an object is often describedherein in terms of identifying and manipulating the objects. Suchmanipulations are virtual manipulations accomplished in the memory orother circuitry/hardware of a computer system. Accordingly, is to beunderstood that embodiments of the present invention may be performedwithin a computer system using data stored within the computer system.

Embodiments described herein provide for the generation of synthesizedx-ray images suitable for training machine learning based systems forperforming various medical image analysis tasks, such as, e.g., noduledetection or lung segmentation. Such synthesized x-ray images aregenerated from CT (computed tomography) images, which are easier toannotate with nodule locations as compared to x-ray images due to theadditional 3D context provided by the CT images. Thus, existing largescale CT datasets with high quality annotations are readily available.However, at times, nodules visible in CT images are not visible in thesynthesized x-ray images, and thus not suitable for utilization intraining machine learning based systems. Embodiments described hereinprovide for the automatic detection of nodules that are visible in theCT images but not visible in the synthesized x-ray images. Accordingly,machine learning based networks may be trained using the synthesizedx-ray images where the nodules are visible, without using thesynthesized x-ray images where the nodules are not visible.

FIG. 1 shows a method 100 for generating synthesized medical images fortraining a machine learning based system, in accordance with one or moreembodiments. The steps of method 100 may be performed by one or moresuitable computing devices, such as, e.g., computer 702 of FIG. 7. Thesynthesized medical images generated according to method 100 may beapplied to train a machine learning based network for performing amedical imaging analysis task, such as, e.g., pulmonary nodule detectionor lung segmentation. In one embodiment, the steps of method 100 may berepeatedly performed for each input medical image in a dataset ofmedical images (e.g., a training dataset).

At step 102, an input medical image in a first modality is received. Theinput medical image may be an image of a chest of a patient showing oneor more pulmonary nodules. The input medical image includes annotationsidentifying the location of the one or more nodules. In one embodiment,the first modality is CT. However, the first modality may be any othersuitable modality, such as, e.g., MRI (magnetic resonance imaging),ultrasound, x-ray, or any other medical imaging modality or combinationsof medical imaging modalities. The input medical image may be a 2D (twodimensional) image or a 3D (three dimensional) volume. The input medicalimage may be received directly from an image acquisition device, suchas, e.g., a CT scanner, as the medical image is acquired, or can bereceived by loading a previously acquired medical image from a storageor memory of a computer system or receiving a medical image that hasbeen transmitted from a remote computer system.

The input medical image comprises a nodule region for each of the one ormore nodules and a remaining region. Each nodule region depicts arespective one of the one or more nodules. The remaining region depictsthe remaining portions of the input medical image that are outside ofthe nodules regions. The nodule regions and the remaining region may bedefined based on the annotations.

FIG. 2A shows an exemplary CT medical image 200, in accordance with oneor more embodiments. CT medical image 200 may be the input medical imagereceived at step 102 of FIG. 1. CT medical image 200 comprises noduleregion 204 depicting nodule 202. CT medical image 200 also comprisesremaining region 206 depicting remaining portions of CT medical image200 outside of nodule region 204.

At step 104, a synthesized medical image in a second modality isgenerated from the input medical image. In one embodiment, the secondmodality is x-ray. However, the second modality may be any othersuitable modality, such as, e.g., CT, MRI, ultrasound, or any othermedical imaging modality or combinations of medical imaging modalities.The synthesized medical image is a translation of the input medicalimage from the first modality to the second modality. Accordingly, thesynthesized medical image also includes the annotation from the inputmedical image, as well as nodule regions and a remaining regioncorresponding to those in the input medical image.

FIG. 2B shows an exemplary synthesized x-ray medical image 210, inaccordance with one or more embodiments. Synthesized x-ray medical image210 may be the synthesized medical image generated at step 104 ofFIG. 1. Synthesized x-ray medical image 210 comprises nodule region 208depicting nodule 202. Synthesized x-ray medical image 210 also comprisesremaining region 210 depicting remaining portions of synthesized x-raymedical image 210 outside of nodule region 208. The locations of noduleregion 208 and remaining region 210 in synthesized x-ray medical image210 respectively correspond to the locations of nodule region 202 andremaining region 106 in CT medical image 200 of FIG. 2A such that therelative location of nodule region 208 and remaining region 210 insynthesized x-ray medical image 210 is the same as the relative locationof nodule region 202 and remaining region 206 in CT medical image 200.

Referring back to FIG. 1, the synthesized medical image may be generatedusing any suitable technique. In one embodiment, a synthesized x-raymedical image is generated from an input CT medical image by projectingthe input CT medical image using Beer's law to approximate the physicalx-ray travelling through tissue. First, the radiodensity of the input CTmedical image (in Hounsfield Units, HU) is transformed to attenuation(cm⁻¹) in order to meaningfully apply Beer's law. Accordingly, Equation(1) is applied for each voxel v:

$\begin{matrix}{{Attenuation} = {{\frac{v}{1000}\left( {\mu_{water} - \mu_{air}} \right)} + \mu_{water}}} & {{Equation}\mspace{14mu}(1)}\end{matrix}$

where the linear attenuation coefficient of water μ_(water)=0.0004 andthe linear attenuation coefficient of air μ_(air)=0.206. The results areclipped such that v≥0. Second, x-ray intensities are computed usingBeer's law to generate a projected x-ray image. The intensities arereduced along the z dimension and then non-linearities are applied.Accordingly, the intensity at pixel p_(xy) is as follows:

p _(xy)=1−exp(Σ_(i) v _(xyi) ·d)  Equation (2)

where v_(xyz) is the voxel at position xyz and d is the length of thevoxel (in centimeters). Finally, the projected x-ray image is furtherprocessed to resemble image processing techniques applied in x-raymachines to generate the synthesized x-ray medical image. In oneembodiment, the contrast of the projected image is reduced by applyinggamma correction (with γ=2.5) and adaptive histogram equalization (withkernel size=128). Exemplary synthesized x-ray medical images are shownin FIG. 3, further described below.

To improve the computational efficiency of projecting the input CTmedical image, some computations may be previously performed offline andthe results saved to memory. In particular, the projection may beseparated into linear and reduction operations. The first two steps inthe projection (up to and including Equation (2)) are performed and theresults of the reduction:

{circumflex over (p)} _(xy)=Σ_(i) v _(xyi)  Equation (3)

are saved to memory. The non-linear operations (e.g., in Equation 2 orthe gamma correction or adaptive histogram equalization) in image spaceare relatively fast and can be computed online. Doing so has theadvantage that the hyperparameters (e.g., γ) may be chosen randomly(e.g., for data augmentation).

It should be understood that the synthesized medical image may also begenerated using any other suitable technique. For example, thesynthesized medical image may be generated from the input medical imageusing a machine learning based network, such as, e.g., a generativeadversarial network.

At step 106, a synthesized nodule image of each of the nodule regionsand a synthesized remaining image of the remaining region are generatedin the second modality. In one example, the synthesized nodule image isa synthesized image of nodule region 202 in FIG. 2A and the synthesizedremaining image is a synthesized image of remaining region 206 withintensity values of pixels in nodule region 202 set to zero in FIG. 2A.

In one embodiment, the synthesized nodule images and the synthesizedremaining image are generated by separately projecting the noduleregions and the remaining region in the input medical image to thesecond modality. In particular, for each nodule i, a separate projectionof each nodule region of the input medical image is performed togenerate synthesized nodule image {circumflex over (n)}_(xy) ^(i). Inaddition, a separate projection of the remaining region of the inputmedical image is performed to generate a synthesized remaining image{circumflex over (r)}_(xy) ^(i), where the intensity values of allpixels of nodules (i.e., all pixels in the nodule regions) are set tozero. The projections may be performed as described with respect to step104. Accordingly, for a input medical image with k nodules, thefollowing identity holds:

{circumflex over (p)} _(xy) ={circumflex over (r)} _(xy)+Σ_(i<k){circumflex over (n)} _(xy) ^(i)  Equation (4)

The synthesized nodule images {circumflex over (n)}_(xy) ^(i) and thesynthesized remaining image {circumflex over (r)}_(xy) ^(i) may be savedto memory, allowing for the dynamic removal and addition of nodules fordata augmentation.

In one embodiment, the synthesized nodule images {circumflex over(n)}_(xy) ^(i) and the synthesized remaining image {circumflex over(r)}_(xy) ^(i) are generated by extracting the synthesized nodule images{circumflex over (n)}_(xy) ^(i) and the synthesized remaining image{circumflex over (r)}_(xy) ^(i) from the synthesized medical image. Thesynthesized nodule images and the synthesized remaining image may alsobe generated using any other suitable technique (e.g., machine learningbased network).

FIG. 3 shows exemplary synthesized images 300, in accordance with one ormore embodiments. Images in column 302 show synthesized medical images{circumflex over (p)}_(xy), images in column 304 show synthesizedremaining images {circumflex over (r)}_(xy) ^(i), and images in column306 show synthesized nodule images Σ{circumflex over (n)}_(xy) ^(i).

Referring back to FIG. 1, in one embodiment, the visibility of thenodules may be increased by highlighting the pixel intensity values ofnodules which are close to the visibility threshold. Accordingly, ahighlighting factor h^(i) may be applied to the synthesized noduleimages to increase the intensity of the pixels in the synthesized noduleimages {circumflex over (n)}_(xy) ^(i), according to Equation (5):

{circumflex over (p)} _(xy) ={circumflex over (r)} _(xy)+Σ_(i<k) h ^(i)·{circumflex over (n)} _(xy) ^(i)  Equation (5)

In one embodiment, the highlighting factor h^(i) may be arbitrarilychosen. For example, a highlighting factor h^(i) may be randomlyselected between [1,2] to generate synthetic medical images with nodulesof a variety of intensities.

FIG. 4 shows a comparison 400 of nodules highlighted with differenthighlighting factors, in accordance with one or more embodiments. Image402 is a synthesized medical image {circumflex over (p)}_(xy), image 404is a synthesized remaining image {circumflex over (r)}_(xy) ^(i), image406 is a synthesized nodule images Σ{circumflex over (n)}_(xy) ^(i),image 408 is a mask of dense anatomical structures that are not visible(as determined at step 108), image 410 is synthesized medical image 402with a highlighting factor of 1, image 412 is synthesized medical image402 with a highlighting factor of 2, image 414 is synthesized medicalimage 402 with a highlighting factor of 5, and image 416 is synthesizedmedical image 402 with a highlighting factor of 10.

At step 108 of FIG. 1, it is determined whether each particular noduleof the one or more nodules is visible in the synthesized medical imagebased on at least one of the synthesized nodule image for the particularnodule and the synthesized remaining image. Nodules visible in the inputmedical image (e.g., a 3D CT image) may be hidden in the synthesizedmedical image (e.g., x-ray image) after projection due to projecting the3D image to 2D space. The synthesized medical image should not includeannotations for nodules that are not visible in the synthesized medicalimage.

One reason that nodules may be hidden in the synthesized medical imagesis that the nodules are hidden behind a dense anatomical structure, suchas, e.g., bones, hearth, liver, etc. In one embodiment, a nodule isconsidered to be hidden behind an anatomical structure in thesynthesized medical image, and thus not visible, when the absoluteintensity of the synthesized remaining image {circumflex over (r)}_(xy)^(i) does not satisfy (e.g., does not exceeds) a certain threshold t(e.g., t=47). Formally, a nodule i with center coordinate (x, y) isconsidered to be not visible in the synthesized medical image if{circumflex over (r)}_(xy) ^(i)≤t and visible in the synthesized medicalimage if {circumflex over (r)}_(xy) ^(i)>t.

FIG. 5 shows exemplary images 500 of hidden nodules hidden behindanatomical structures, in accordance with one or more embodiments.Images in column 502 show synthesized medical images {circumflex over(p)}_(xy), images in column 504 show synthesized remaining images{circumflex over (r)}_(xy) ^(i), images in column 506 show synthesizednodule images Σ{circumflex over (n)}_(xy) ^(i), and images in column 508show a mask of dense anatomical structures, where the light portions ofthe mask represent dense anatomical structure. As shown in the images incolumn 508, nodules are hidden behind anatomical structures (e.g., thehearth and the liver).

Another reason that nodules may be hidden in the synthesized medicalimage is that the relative intensity of the nodule tissue is lowcompared to the intensity of the tissue in surrounding area. In oneembodiment, the nodule is considered visible if two conditions are met.First, a change between an intensity at the center of the nodules in thesynthesized remaining image and an intensity at the center of thenodules in the synthesized nodule images increases by at least a certainthreshold t (e.g., 15%). Formally, for a nodule i with center coordinate(x, y) with threshold t=15%, the equation

$\frac{{\hat{r}}_{xy}^{i} + {\hat{n}}_{xy}^{i}}{{\hat{r}}_{xy}^{i}} \geq {1.15.}$

Second, a change between an average intensity of pixels within half ofthe radius of the nodules in the synthesized remaining image and anaverage intensity of pixels within half of the radius of the nodules inthe synthesized nodule image increases by at least a certain threshold t(e.g., 10%) amount.

FIG. 6 shows exemplary images 600 of hidden nodules hidden due to therelative intensity of the nodules, in accordance with one or moreembodiments. Images in column 602 show synthesized medical images{circumflex over (p)}_(xy), images in column 604 show synthesizedremaining images {circumflex over (r)}_(xy) ^(i), images in column 606show synthesized nodule images Σ{circumflex over (n)}_(xy) ^(i), andimages in column 608 show a mask of thresholded pixel intensity. Asshown in the images in column 608, the nodules are hidden due to the lowrelative intensity of the nodules as compared to the tissue in thesurrounding area.

At step 110 of FIG. 1, in response to determining that at least onenodule of the one or more nodules is not visible in the synthesizedmedical image, the annotation for the at least one not visible nodule isremoved from the synthesized nodule image.

At step 112, the synthesized medical image is output. For example, thesynthesized medical image can be output by displaying the synthesizedmedical image on a display device of a computer system, storing thesynthesized medical image on a memory or storage of a computer system,or by transmitting the synthesized medical image to a remote computersystem.

In one embodiment, synthesized medical images generated according tomethod 100 may be applied for training machine learning based networksfor medical image analysis tasks, such as, e.g., nodule detection ornodule segmentation. For example, a nodule detector network implementedby a 2D Faster R-CNN (convolutional neural network) or a lungsegmentation network implemented by a 2D U-Net may be respectivelytrained for nodule detection or lung segmentation in x-ray images usingsynthesized x-ray medical images generated according to method 100. Inone embodiment, such networks are trained using only the synthesizedx-ray medical images. However, in another embodiment, such networks aretrained using a combination of synthesized x-ray medical images and realx-ray medical images. Advantageously, the synthesized x-ray medicalimages provides additional examples and examples of difficult to detectnodules in the training dataset, while having real x-ray medical imagesin the training dataset ensures that the feature extractors in the earlylayers of the networks are well adapted to the data distribution foundin real x-ray medical images.

In one embodiment, various nodule augmentation techniques may be appliedto generate the synthesized medical image in method 100.

In one embodiment, a nodule may be positioned in the synthesized medicalimage at step 104 by utilizing a database of previously acquiredsynthesized nodule images. A suitable position of the nodule may bedetermined based on visibility considerations described herein as wellas a lung segmentation. Nodules with an intensity of less than, e.g.,0.5 (in linear space) may be removed from the database since there areno positions where they are visible.

In another embodiment, the input medical image received at step 102 maybe rotated before generating the synthesized medical image at step 104.By rotating the input medical image, a variety of different viewpointangles of the nodules are obtained, multiplying the amount of relevantfeatures available per nodule. The rotations are performed during aprior offline stage, since they are operations in 3D space. To make thiscomputationally feasible, a limited number (e.g., 20) of rotations pernodule are performed and the corresponding synthesized medical imagesare independently saved as separate images. During application of thesynthesized medical images (e.g., during training of a machine learningbased network), one of the images may be selected (e.g., randomly).

In another embodiment, after the synthesized medical image is generatedat step 104, the nodules may be resized. In particular, a bi-cubicup-sampling or down-sampling of the synthesized medical image isapplied. To account for the changes of tissue size in the z dimension,all synthesized medical images may be multiplied by a resize factor.This follows Beer's law since the ray has to travel through more tissue.

In another embodiment, the input medical image received at step 102 maybe generated to include artificial or synthesized nodules as describedin U.S. patent application Ser. No. 16/445,435, filed Jun. 19, 2019, andU.S. patent application Ser. No. 16/570,214, filed Sep. 13, 2019, thedisclosures of which are incorporated herein by reference in theirentirety. Synthesizing nodules in 3D space (e.g., of a CT image) iseasier than in 2D space (e.g., an x-ray image).

Systems, apparatuses, and methods described herein may be implementedusing digital circuitry, or using one or more computers using well-knowncomputer processors, memory units, storage devices, computer software,and other components. Typically, a computer includes a processor forexecuting instructions and one or more memories for storing instructionsand data. A computer may also include, or be coupled to, one or moremass storage devices, such as one or more magnetic disks, internal harddisks and removable disks, magneto-optical disks, optical disks, etc.

Systems, apparatus, and methods described herein may be implementedusing computers operating in a client-server relationship. Typically, insuch a system, the client computers are located remotely from the servercomputer and interact via a network. The client-server relationship maybe defined and controlled by computer programs running on the respectiveclient and server computers.

Systems, apparatus, and methods described herein may be implementedwithin a network-based cloud computing system. In such a network-basedcloud computing system, a server or another processor that is connectedto a network communicates with one or more client computers via anetwork. A client computer may communicate with the server via a networkbrowser application residing and operating on the client computer, forexample. A client computer may store data on the server and access thedata via the network. A client computer may transmit requests for data,or requests for online services, to the server via the network. Theserver may perform requested services and provide data to the clientcomputer(s). The server may also transmit data adapted to cause a clientcomputer to perform a specified function, e.g., to perform acalculation, to display specified data on a screen, etc. For example,the server may transmit a request adapted to cause a client computer toperform one or more of the steps or functions of the methods andworkflows described herein, including one or more of the steps orfunctions of FIG. 1. Certain steps or functions of the methods andworkflows described herein, including one or more of the steps orfunctions of FIG. 1, may be performed by a server or by anotherprocessor in a network-based cloud-computing system. Certain steps orfunctions of the methods and workflows described herein, including oneor more of the steps of FIG. 1, may be performed by a client computer ina network-based cloud computing system. The steps or functions of themethods and workflows described herein, including one or more of thesteps of FIG. 1, may be performed by a server and/or by a clientcomputer in a network-based cloud computing system, in any combination.

Systems, apparatus, and methods described herein may be implementedusing a computer program product tangibly embodied in an informationcarrier, e.g., in a non-transitory machine-readable storage device, forexecution by a programmable processor; and the method and workflow stepsdescribed herein, including one or more of the steps or functions ofFIG. 1, may be implemented using one or more computer programs that areexecutable by such a processor. A computer program is a set of computerprogram instructions that can be used, directly or indirectly, in acomputer to perform a certain activity or bring about a certain result.A computer program can be written in any form of programming language,including compiled or interpreted languages, and it can be deployed inany form, including as a stand-alone program or as a module, component,subroutine, or other unit suitable for use in a computing environment.

A high-level block diagram of an example computer 702 that may be usedto implement systems, apparatus, and methods described herein isdepicted in FIG. 7. Computer 702 includes a processor 704 operativelycoupled to a data storage device 712 and a memory 710. Processor 704controls the overall operation of computer 702 by executing computerprogram instructions that define such operations. The computer programinstructions may be stored in data storage device 712, or other computerreadable medium, and loaded into memory 710 when execution of thecomputer program instructions is desired. Thus, the method and workflowsteps or functions of FIG. 1 can be defined by the computer programinstructions stored in memory 710 and/or data storage device 712 andcontrolled by processor 704 executing the computer program instructions.For example, the computer program instructions can be implemented ascomputer executable code programmed by one skilled in the art to performthe method and workflow steps or functions of FIG. 1. Accordingly, byexecuting the computer program instructions, the processor 704 executesthe method and workflow steps or functions of FIG. 1. Computer 702 mayalso include one or more network interfaces 706 for communicating withother devices via a network. Computer 702 may also include one or moreinput/output devices 708 that enable user interaction with computer 702(e.g., display, keyboard, mouse, speakers, buttons, etc.).

Processor 704 may include both general and special purposemicroprocessors, and may be the sole processor or one of multipleprocessors of computer 702. Processor 704 may include one or morecentral processing units (CPUs), for example. Processor 704, datastorage device 712, and/or memory 710 may include, be supplemented by,or incorporated in, one or more application-specific integrated circuits(ASICs) and/or one or more field programmable gate arrays (FPGAs).

Data storage device 712 and memory 710 each include a tangiblenon-transitory computer readable storage medium. Data storage device712, and memory 710, may each include high-speed random access memory,such as dynamic random access memory (DRAM), static random access memory(SRAM), double data rate synchronous dynamic random access memory (DDRRAM), or other random access solid state memory devices, and may includenon-volatile memory, such as one or more magnetic disk storage devicessuch as internal hard disks and removable disks, magneto-optical diskstorage devices, optical disk storage devices, flash memory devices,semiconductor memory devices, such as erasable programmable read-onlymemory (EPROM), electrically erasable programmable read-only memory(EEPROM), compact disc read-only memory (CD-ROM), digital versatile discread-only memory (DVD-ROM) disks, or other non-volatile solid statestorage devices.

Input/output devices 708 may include peripherals, such as a printer,scanner, display screen, etc. For example, input/output devices 708 mayinclude a display device such as a cathode ray tube (CRT) or liquidcrystal display (LCD) monitor for displaying information to the user, akeyboard, and a pointing device such as a mouse or a trackball by whichthe user can provide input to computer 702.

An image acquisition device 714 can be connected to the computer 702 toinput image data (e.g., medical images) to the computer 702. It ispossible to implement the image acquisition device 714 and the computer702 as one device. It is also possible that the image acquisition device714 and the computer 702 communicate wirelessly through a network. In apossible embodiment, the computer 702 can be located remotely withrespect to the image acquisition device 714.

Any or all of the systems and apparatus discussed herein may beimplemented using one or more computers such as computer 702.

One skilled in the art will recognize that an implementation of anactual computer or computer system may have other structures and maycontain other components as well, and that FIG. 5 is a high levelrepresentation of some of the components of such a computer forillustrative purposes.

The foregoing Detailed Description is to be understood as being in everyrespect illustrative and exemplary, but not restrictive, and the scopeof the invention disclosed herein is not to be determined from theDetailed Description, but rather from the claims as interpretedaccording to the full breadth permitted by the patent laws. It is to beunderstood that the embodiments shown and described herein are onlyillustrative of the principles of the present invention and that variousmodifications may be implemented by those skilled in the art withoutdeparting from the scope and spirit of the invention. Those skilled inthe art could implement various other feature combinations withoutdeparting from the scope and spirit of the invention.

1. A computer implemented method comprising: receiving an input medicalimage in a first modality, the input medical image comprising a noduleregion for each of one or more nodules and a remaining region, the inputmedical image comprising an annotation for each of the one or morenodules; generating, from the input medical image, a synthesized medicalimage in a second modality, the synthesized medical image comprising theannotation for each of the one or more nodules; generating, in thesecond modality, 1) a synthesized nodule image of each of the noduleregions and 2) a synthesized remaining image of the remaining region;determining whether each particular nodule of the one or more nodules isvisible in the synthesized medical image based on at least one of thesynthesized nodule image for the particular nodule and the synthesizedremaining image; and in response to determining that at least one noduleof the one or more nodules is not visible in the synthesized medicalimage, removing the annotation for the at least one not visible nodulefrom the synthesized nodule image.
 2. The method of claim 1, whereindetermining whether each particular nodule of the one or more nodules isvisible in the synthesized medical image based on at least one of thesynthesized nodule image for the particular nodule and the synthesizedremaining image comprises: comparing an intensity of pixels in thesynthesized remaining image to a threshold.
 3. The method of claim 1,wherein determining whether each particular nodule of the one or morenodules is visible in the synthesized medical image based on at leastone of the synthesized nodule image for the particular nodule and thesynthesized remaining image comprises: determining a change between anintensity at a center of the particular nodule in the synthesizedremaining image and an intensity at the center of the particular nodulein the synthesized nodule image; and comparing the intensity change to athreshold.
 4. The method of claim 1, wherein determining whether eachparticular nodule of the one or more nodules is visible in thesynthesized medical image based on at least one of the synthesizednodule image for the particular nodule and the synthesized remainingimage comprises: determining a change between an average intensity ofpixels within half of a radius of the particular nodule in thesynthesized remaining image and an average intensity of pixels withinhalf of the radius of the particular nodule in the synthesized noduleimage; and comparing the change to a threshold.
 5. The method of claim1, wherein generating, from the input medical image, a synthesizedmedical image in a second modality comprises: increasing an intensity ofpixels in the synthesized nodule images by a highlighting factor.
 6. Themethod of claim 1, wherein generating, from the input medical image, asynthesized medical image in a second modality comprises: positioning apreviously acquired synthesized nodule image in the synthesized medicalimage.
 7. The method of claim 1, wherein generating, from the inputmedical image, a synthesized medical image in a second modalitycomprises: rotating the input medical image; and generating thesynthesized medical image from the rotated input medical image.
 8. Themethod of claim 1, further comprising: training a machine learning basednetwork based on the synthesized medical image.
 9. The method of claim1, wherein receiving an input medical image in a first modalitycomprises: generating the input medical image to include one or moresynthesized nodules.
 10. The method of claim 1, wherein the firstmodality is computed tomography and the second modality is x-ray.
 11. Anapparatus comprising: means for receiving an input medical image in afirst modality, the input medical image comprising a nodule region foreach of one or more nodules and a remaining region, the input medicalimage comprising an annotation for each of the one or more nodules;means for generating, from the input medical image, a synthesizedmedical image in a second modality, the synthesized medical imagecomprising the annotation for each of the one or more nodules; means forgenerating, in the second modality, 1) a synthesized nodule image ofeach of the nodule regions and 2) a synthesized remaining image of theremaining region; means for determining whether each particular noduleof the one or more nodules is visible in the synthesized medical imagebased on at least one of the synthesized nodule image for the particularnodule and the synthesized remaining image; and means for in response todetermining that at least one nodule of the one or more nodules is notvisible in the synthesized medical image, removing the annotation forthe at least one not visible nodule from the synthesized nodule image.12. The apparatus of claim 11, wherein the means for determining whethereach particular nodule of the one or more nodules is visible in thesynthesized medical image based on at least one of the synthesizednodule image for the particular nodule and the synthesized remainingimage comprises: means for comparing an intensity of pixels in thesynthesized remaining image to a threshold.
 13. The apparatus of claim11, wherein the means for determining whether each particular nodule ofthe one or more nodules is visible in the synthesized medical imagebased on at least one of the synthesized nodule image for the particularnodule and the synthesized remaining image comprises: means fordetermining a change between an intensity at a center of the particularnodule in the synthesized remaining image and an intensity at the centerof the particular nodule in the synthesized nodule image; and means forcomparing the intensity change to a threshold.
 14. The apparatus ofclaim 11, wherein the means for determining whether each particularnodule of the one or more nodules is visible in the synthesized medicalimage based on at least one of the synthesized nodule image for theparticular nodule and the synthesized remaining image comprises: meansfor determining a change between an average intensity of pixels withinhalf of a radius of the particular nodule in the synthesized remainingimage and an average intensity of pixels within half of the radius ofthe particular nodule in the synthesized nodule image; and means forcomparing the change to a threshold.
 15. The apparatus of claim 11,wherein the means for generating, from the input medical image, asynthesized medical image in a second modality comprises: means forincreasing an intensity of pixels in the synthesized nodule images by ahighlighting factor.
 16. A non-transitory computer readable mediumstoring computer program instructions, the computer program instructionswhen executed by a processor cause the processor to perform operationscomprising: receiving an input medical image in a first modality, theinput medical image comprising a nodule region for each of one or morenodules and a remaining region, the input medical image comprising anannotation for each of the one or more nodules; generating, from theinput medical image, a synthesized medical image in a second modality,the synthesized medical image comprising the annotation for each of theone or more nodules; generating, in the second modality, 1) asynthesized nodule image of each of the nodule regions and 2) asynthesized remaining image of the remaining region; determining whethereach particular nodule of the one or more nodules is visible in thesynthesized medical image based on at least one of the synthesizednodule image for the particular nodule and the synthesized remainingimage; and in response to determining that at least one nodule of theone or more nodules is not visible in the synthesized medical image,removing the annotation for the at least one not visible nodule from thesynthesized nodule image.
 17. The non-transitory computer readablemedium of claim 16, wherein generating, from the input medical image, asynthesized medical image in a second modality comprises: positioning apreviously acquired synthesized nodule image in the synthesized medicalimage.
 18. The non-transitory computer readable medium of claim 16,wherein generating, from the input medical image, a synthesized medicalimage in a second modality comprises: rotating the input medical image;and generating the synthesized medical image from the rotated inputmedical image.
 19. The non-transitory computer readable medium of claim16, further comprising: training a machine learning based network basedon the synthesized medical image.
 20. The non-transitory computerreadable medium of claim 16, wherein the first modality is computedtomography and the second modality is x-ray.